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Abstract 

The National Survey of Student Engagement (NSSE) has been used at universities across the U.S. and Canada to 
gather information about the quality of engagement of first-year students and graduating students. Institutions 
use NSSE’s five benchmarks of effective educational practice to compare themselves with other schools and to 
focus in on ways to improve the educational experiences of their students. However, studies indicate that these 
benchmarks may not be a valid way to convey NSSE information. This study was conducted to investigate 
whether or not NSSE’s five-factor model is the best fit for student engagement data collected at a large, public, 
research-intensive, land-grant university. The five-factor model did not fit the data for the 2008 sample of senior 
students at this university. Rather, a revised model using six factors instead of five and 21 of 42 items provided a 
more valid test blueprint. This new model was then tested and found to fit the 2011 sample of senior students at 
the same university. Discussion regarding use of a nationally collected data at an individual institution is 
provided. 

Keywords: National Survey of Student Engagement (NSSE), exploratory factor analysis(EFA), confirmatory 
factor analysis (CFA), student engagement 

1. Introduction 

Ever since Astin (1984) proposed a developmental model of college student learning that emphasized the 
concept of involvement, educators in higher education around the globe have become more focused on 
developing what has come to be known as “student engagement.” Through the curriculum, faculty have 
developed “learning communities,” service learning opportunities, internships, and global exchanges as ways to 
augment “the amount of physical and psychological energy” (Astin, 1984, p. 297) students devote to their 
academic experiences. In addition, universities have developed co-curricular activities intended to increase 
students’ interactions with peers and others associated with the university community. Increases in student 
engagement have been shown to be positively linked to individual satisfaction, personal growth, and the overall 
success of students (Astin, 1993; Bruffee, 1993; Goodsell, Maher, & Tinto, 1992; McKeachie, Pintrich, Lin, & 
Smith, 1986; Pascarella & Terenzini, 1991). 

Broadly defined, student engagement is related to student motivation and commitment to learning, a sense of 
belonging, and development of relationships with adults, peers, and others who support their learning and 
accomplishments (Jones, 2009). More specifically, it is the “time and energy students devote to educationally 
sound activities inside and outside of the classroom” (Kuh, 2003, p. 25). Students who are more engaged are 
more likely to be academically successful (Pascarella &Terenzini, 2005; Tinto, 1997), more likely to graduate, 
receive better grades, and be more satisfied with their collegiate experience (Astin & Sax, 1998; Kuh, Kinzie, 
Schuh, Whitt, & Associates, 2005). 

Since 2000, the most often used instrument to measure higher education students’ perceptions of engagement is 
the National Survey of Student Engagement (NSSE). The NSSE was specifically created to examine the degree 
to which students engage in meaningful educational practices that are significantly related to students’ positive 
outcomes (Kuh, 2003). It is administered every spring semester to first-year students and seniors at colleges and 
universities in the U.S. and Canada. Students from almost 1,500 different baccalaureate-granting institutions 
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have taken the NSSE. In 2011 alone, almost two million first-year and senior students from 751 colleges and 
universities were invited to participate in the NSSE administration (NSSE, 2011). 

NSSE provides summative information on five scales that can be used as benchmarks to compare institutions to 
institutions or cohorts within institutions. More and more schools have become involved with the NSSE and in 
comparing themselves with other universities on these scales (Lutz & Culver, 2010). The results have also been 
reported as part of the college ranking processes in magazines and newspapers, such as USA Today , and in 
centralized information websites, such as the College Portrait (http://www.collegeportraits.org/), sponsored by 
the Voluntary System of Accountability. Given the high profile nature of this reporting, the results of 
comparisons on the five benchmark scales of the NSSE have raised the stakes of this type of assessment. 

Though Kuh (2009) emphasizes that the NSSE project was intended to “provide high-quality, actionable data 
that institutions can use to improve the undergraduate experience” (p. 9), there have been questions raised about 
how well those data capture the perceived experiences of students at a particular university. For example, 
Swerdzewski, Miller, and Mitchell (2007) collected evidence to determine whether or not NSSE information 
could be used to make “programmatic and policy changes” (p. 9) at James Madison University (VA). Their 
results indicated that a confirmatory factor analysis of the benchmarks specified by the NSSE literature produced 
poor model fit. Their conclusion was that a comparison of benchmark scores from their sample of students to 
scores from a sample at another university should not be made. They further noted that policy or programmatic 
decisions should not be made at that particular university based on the benchmark scores. 

In a similar study. Lutz and Culver (2010) analyzed the data from a large land-grant university in the 
southeastern U.S. Using confirmatory factor analyses (CFA) on data from all students at the institution and then 
on data from first-year and seniors, respectively, they found that none of the indices used, such as Chi-Square, 
Comparative Fit Index (CFI), Tucker Louis Index (TLI), Root Mean Square Error of Approximation (RMSEA), 
and Standardized Root Mean Square Residual (SRMR) supported the model for any of the groups at that 
particular university. As with Swerdzewski, Miller, and Mitchell (2007), they concluded that assessment 
practitioners should be cautious in their use of NSSE data at their institution. 

In yet another validation study conducted by LaNasa, Cabrera, and Trangsrud (2009) on only the first-year 
cohort at their institution, the five-factor NSSE model failed to fit the data collected from their first-year student 
cohort. On further analyses, they found that an eight factor model provided a better fit to their data. Similar 
results obtained by Campbell and Cabrera (2011) at a research-intensive university led them to conclude that 
carefully examining the reliability and validity of the NSSE benchmarks at the institutional level is needed 
before any school uses the data collected. 

In his effort to provide valid data from the instrument. Pike (2006) made a strong case for using 12 NSSE scale 
lets, rather than the five benchmarks. His justification for developing scale lets was that they help overcome 
difficulties in disaggregating survey data at the college or department level. In addition, they allow decision 
makers at the institution and system level to gauge experiences of institution-specific student engagement, and 
they assist in detecting areas that need rectification. Furthermore, scale let scores provided greater explanatory 
power than the five NSSE benchmark scale scores. 

In summary, there are questions about the validity of the five benchmark subscale scores obtained from the 
NSSE. Furthermore, these scores have been presented as applicable to all types of institutions, regardless of 
mission, location, or type of student served. However, previous research has indicated that levels of engagement, 
and types of engagement, may vary by such institutional factors, not to mention personal characteristics of the 
student. For instance. Pike, Kuh, and Gonyea (2003) found that students attending different types and sizes of 
colleges and universities reported having different patterns of experience in college. Characteristics of schools 
also play a factor. For instance, colleges with larger percentages of students living on-campus tend to have 
higher levels of engagement (Pike & Kuh, 2005). Smart, Feldman, and Ethington (2000) have also suggested 
that disciplinary area may also be related to student engagement. In addition. Harper (2009) has suggested that 
racial minorities engage differently with their college environments, making measurement of this engagement 
more complex. 

2. Purpose of the Study 

The purpose of this study was to examine whether the five NSSE benchmark scales provided valid information 
for use at our particular institution. Such an examination is necessary because of the concerns as noted in the 
literature. Other studies have been conducted at research-intensive universities (e.g., Campbell &Cabrera, 2011; 
LaNasa, Cabrera, & Trangsrud, 2009), but both of these were urban institutions with different missions than our 
public, land-grant institution. Though the items making up the NSSE have strong theoretical grounding, there are 
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questions regarding the construct validity of the five NSSE benchmarks. In addition, good assessment practice 
suggests caution in using institution-specific reports derived from a national survey such as NSSE without 
ensuring their validity on the local level ( Yarbrough, Shulha, Hopson, & Caruthers, 2011). Specifically, we seek 
to answer three research questions related to the construct validity of the NSSE five-benchmark model using the 
student data from seniors at our institution. Our focus was on the senior cohort only because of their longer 
periods of time spent at our institution. 

1. Using the 2008 senior cohort sample, are there five separate, stable benchmarks that measure 
engagement at our institution? 

2. If these five benchmarks do not provide a valid model of engagement with this sample, is there 
another model that provides a better fit for the data? 

3. Using the best model, either the NSSE five-benchmark or the better-fit model, can we provide a valid 
measure of student engagement for the 2011 senior cohort sample at this institution? 

3. Methods 

3.1 Samples 

Sample 1. The NSSE data used to answer research questions #1 and #2 were collected in the spring of 2008. A 
total of 757 seniors at Virginia Tech responded to the web-based version of the NSSE instrument, yielding a 
response rate of 39%. However, after those with missing data were removed, the sample was 679. Of these, 56% 
were male students; 44% female. In terms of race, 6.6% were Asian, Asian American or Pacific Islander; 2.5% 
were African American; 81.3% were White (non-Hispanic), and 9.6% indicated “other.” Further, 3.5% 
identified as international students. 

Sample 2. The NSSE data used to answer research question #3 were collected in the spring of 2011. A total of 
756 seniors responded to the web-based NSSE instrument, yielding a response rate of 17%. However, after those 
with missing data were removed, the sample was 655. Of these, 49.2% were male; 50.8% were female. Asian, 
Asian American, or Pacific Islanders constituted 6.9% of the sample; 2.4% were African American; 82.4% were 
White (non-Hispanic), and 8.3% indicated “other.” International students constituted 2.7% of the sample. 

3.2 Instrument 

The NSSE’s development began with the Pew Foundation’s effort to bring together a panel of experts to develop 
a survey to measure student engagement under the accepted theory that more engagement means more learning 
(see Chickering & Gamson, 1987; NSSE, 2008). The NSSE instrument collects information in five categories. 
First, students are asked about their participation in “dozens of educationally purposeful activities” (Kuh, 2009), 
such as interacting with faculty and with peers, the amount of time they spend studying or in other activities. A 
second set of questions asks about the amount of reading and writing they do, as well as questions about their 
coursework and examinations. A third set of questions asks students about their college environment, whether 
they feel supported and the quality of their interactions with others (peers, faculty, and administrators) on 
campus. A fourth grouping of questions asks students for their background information, such as age, gender, 
race/ethnicity, and major field of study. A fifth set of questions asks students to estimate their educational and 
personal growth since beginning college in such areas as written and oral communication, ethical development, 
and vocational preparation. 

Five benchmarks were created from 42 key items on the NSSE survey using a combination of engagement and 
educational impact theory and exploratory factor analysis (Pike, Kuh, McCormick, Ethington, & Smart, 2011). 
The five benchmarks were designed to tap into five different, though inter-related constructs of student 
engagement with their institution and to be applicable to any institutional setting, regardless of size, mission, or 
students served (Campbell & Cabrera, 2011). These benchmarks, labeled the Benchmarks of Effective 
Educational Practice, are 

• Level of Academic Challenge (LAC) 

• Active and Collaborative Learning (ACL) 

• Enriching Educational Experiences (EEE) 

• Student-Faculty Interaction (SFI) 

• Supportive Campus Environment (SCE) 

The Level of Academic Challenge (LAC) was measured by 11 items, such as a number of assigned textbooks, 
number of written papers or reports, emphasis of coursework (analysis, synthesis, making of judgments, 
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application), and whether the campus environment supports spending significant amounts of time on academic 
work. Active and Collaborative Learning (ACL) was measured by seven items, such as how often students asked 
questions in class, made a class presentation, worked with classmates outside of class, or participated in a 
community-based project as part of a class. Enriching Educational Experiences (EEE) was measured by 12 items, 
such as participation in co-curricular activities, internships, community service, foreign language coursework, or 
learning communities. Student-Faculty Interactions (SFI) was measured by six items, such as how often students 
discussed grades or assignments with an instructor, talked about career plans with a faculty member, or worked 
with faculty members on activities or research outside of class. Finally, the Supportive Campus Environment 
(SCE) was measured by six items, such as quality of relationships with faculty members, other students, 
administrative personnel, and whether the campus environment provides support for thriving socially. 

3.3 Model Testing 

Using the data from the 2008 senior sample, we tested a confirmatory factor model with the NSSE’s five 
benchmark scales as each of the factors using Maximum Likelihood (ML) methods in Mplus version 5.1 
(Muthen & Muthen, 2008). There are many measures for fit, but we relied on three robust measures of fit to 
judge the CFA model: the Comparative Fix Index (CFI), the Tucker-Lewis Index (TLI), and the Root Mean 
Square Error of Approximation (RMSEA). Guided by recommendations from the literature (Hu & Bender, 1999; 
Yu, 2002), we sought CFI and TLI values of 0.95 or higher to indicate an appropriate level of fit. In regard to 
RMSEA values (see Brown, 2006; Hu & Bender, 1999), we considered values less than .06 to be suitable. 
Close fit (CFit) indicates the probability (p) that the RMSEA is less than or equal to 0.05 (see Brown, 2006). We 
also report the Standardized Root Mean Square Residual (SRMR), with a value of .08 or less indicated an 
acceptable model (Hu & Bender, 1999). 

After the NSSE five-benchmark structure did not provide a good fit for the student engagement data at our 
institution, we then employed exploratory factor analysis (EFA), using promax rotation, to identify a factor 
structure that might provide a better fit for the 2008 senior data from our university. This new model was then 
tested, using confirmatory factor analysis on the data from the 2011 senior sample group at the same institution. 

4. Results 

4.1 Confirmatory Factor Analysis Results for NSSE Five-factor Model 

The results of the confirmatory factor analysis of the NSSE five-factor model indicated that the model was not a 
good fit for the 2008 senior data at our institution. Table lpresents the reliability information of the original 
instrument and its five factors, as well as the factor loadings for each of the 42 items. As shown, the overall 
reliability was .869. Though Cronbach’s alpha of each of the five factors (LAC, ACL, SFI, EEE, and SCE) 
ranged between .616 and .772, many of the items did not load well on the predicted factor. For example, of the 
11 items predicted to load on Level of Academic Engagement, 5 of them had loadings below .3 and two more 
had loadings above .3 but less than .5. Similarly, of the 12 items predicted to load on Enriching Educational 
Experiences, nine had loadings below .3. The only factor to have all predicted items load above .5 was 
Supportive Campus Environment. 


Table 1. Benchmarks and Item loadings for the NSSE Five-Benchmark Model With the 2008 Senior Sample 


(n = 679) 

Benchmark Measure 

Loading 

Benchmark Reliability 

Original Instrument (Overall scale) 


.869 

Level of Academic Challenge 


.742 

READASGN 

.247 


WRITEMOR 

.295 


WRITEMID 

.318 


WRITESML 

.218 


ANALYZE 

.705 


SYNTHESZ 

.725 
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EVALUATE .706 

APPLYING .688 

WORKHARD .450 

ACADPR01 .273 

ENVSCHOL .292 

Active and Collaborative Learning 

CLQUEST .558 

CLPRESEN .477 

CLASSGRP .312 

OCCGRP .385 

TUTOR .437 

COMMPROJ .469 

OOCIDEAS .603 

Student-Faculty Interaction 

FACGRADE .580 

FACPLANS .709 

FACIDEAS .735 

FACFEED .502 

FACOTHER .649 

RESRCH04 .314 

Enriching Educational Experiences 

DIVRSTUD .830 

DIFFSTU2 .842 

ENVDIVRS .236 

COCURR01 .194 

ITACADEM .301 

INTERN04 .138 

VOLNTR04 .231 

LRNCOM04 .259 

FORLNG04 .134 

STDABR04 .185 

INDSTD04 .112 

SNRX04 .158 

Supportive Campus Environment 

ENVSOCAL .603 

ENVSUPRT .612 

ENVACAD .638 

ENVSTU .511 

ENVFAC .724 

ENVADM .628 


.723 


.671 


.616 


.772 


Note: Refer to the College Student Report NSSE 2008 and 2011 Codebook for the actual questions 


(www.nsse.iub.edu/). 
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The fit indices presented in Table 2 further demonstrate that the NSSE five-factor model does not provide an 
adequate fit for the 2008 data collected from seniors at our institution. For example, the CFI is below our .95 
threshold and the RMSEA is above .06, with the Close fit (CFit) indicating a probability of zero that RMSEA is 
less than or equal to 0.05 (see Brown, 2006). 


Table 2. Fit Indices of the NSSE Five-Benchmark Model With the 2008 Senior Sample 


Cohort 

CFI 

TFI 

RMSEA 

CFit 

SRMR 

Seniors 

.663 

.641 

.068 

0 

.077 


4.2 Results of Exploratory Factor Analysis 

Given the results of our CFA and the failure of the NSSE five-factor model to fit our data, we then used 
exploratory factor analysis (EFA) to explore the factor structure that would emerge from the data. Using 
principal components analysis and retaining factors of eigenvalues greater than 1, the initial results revealed 11 
potential factors after promax rotation. Item loadings on the factors were then examined and items with 
cross-loadings on factors were eliminated from the analyses. For the purposes of our study, items were retained 
if their primary factor loading was at least .50 with no cross-loading of .30 or above. Through this process, 21 
items were removed, and the principal components factor analysis using promax rotation was run on the 
remaining 21 items. This analysis resulted in six factors, explaining 61.89% of the variance. These factors we 
have labeled(l) student-faculty interaction (SFI), with six items and a Cronbach’s reliability of .751 ;(2) 
higher-order thinking skill (HOTS), with four items and a reliability of .817;(3) supportive campus environment 
(SCE), with four items and a reliability of .795;(4) quality of relationship (QoR), with three items and a 
reliability of .705;(5) writing challenge (WC), with three items and reliability of .661;(6) diversity (D), with two 
items and a reliability of .850. 


Table 3. Factor Foadings for Exploratory Factor Analysis With Promax Rotation of the Revised NSSE Model 
With the 2008 Senior Sample (n = 679) 



SFI 

HOTS 

SCE 

Component 

QoR WC 

D 

writemor 




.707 


writemid 




.847 


writesml 




.794 


analyze 


.841 




synthesz 


.849 




evaluate 


.769 




applying 


.774 




clquest 

.492 


-.296 

.222 


facgrade 

.453 




.245 

facplans 

.699 





facideas 

.720 





facother 

.784 





resrch04 

.709 





divrstud 





.938 

diffstu2 





.909 

envdivrs 



.805 



envsocal 



.819 



envsuprt 



.535 

.300 
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envnacad 

envstu 

envfac .251 

envadm 


.785 

.724 

.774 

.840 


% of variance 26.01% 9.86% 8.10% 7.25% 5.71% 4.96% 

Explained 

Note. Factor loadings < .2 are suppressed. Bold numbers indicate primary factor 
loading. SFI = Student-Faculty Interaction; HOTS = Higher-Order Thinking Skill; 
SCE = Supportive Campus Environment; QoR = Quality of Relationship; WC = 
Writing Challenge; D = Diversity. 


4.3 Results of Confirmatory Factor Analysis of Six-factor Model 

With this new model, a CFA was conducted on the 2011 seniors’ data to validate this new test blueprint. Table 4 
presents the results of the CFA, including the factor loadings for each of the items. As shown, the overall 
reliability was .827, with strong reliabilities on all of the scales: writing challenge (.616), higher-order thinking 
skills (.779), student-faculty interaction (.715), diversity (.822), supportive campus environment (.807), and 
quality of relationships (.683). 


Table 4. Benchmarks and Item loadings for the Revised NSSE Six-Benchmark Model With the 2011 Senior 
Sample (n = 655) 


Benchmark Measure Foading 

Items 

Original Instrument (Overall scale) 


Writing Challenge 

WRITEMOR .443 

WRITEMID .966 

WRITESML .496 

Higher-Order Thinking Skill 

ANALYZE .646 

YNTHESZ .829 

VALUATE .659 

APPLYING .615 

Student-Faculty Interaction 

CLQUEST .490 

FACGRADE .568 

FACPLANS .720 

FACIDEAS .666 

FACOTHER .588 

RESRCH04 .297 

Diversity 

DIVRSTUD .816 

DIFFSTU2 .856 

Supportive Campus Environment 

ENVDIVRS .722 


Benchmark Reliability 


.827 

.616 


.779 


.715 


.822 


.807 
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ENVSOCAL .719 

ENVSUPRT .640 

ENVACAD .792 

Quality of Relationships 

ENVSTU .506 

ENVFAC .873 

ENVADM .628 


.683 


Table 5 presents the model fit indices of this CFA. As shown, there was a good model-data fit between the new 
model and the 2011 seniors’ data. In addition to other strong fit indices, the RMSEA of .038 is less than our 
threshold, indicating a good fit, with the CFit of 1.0 indicating a strong probability that the RMSEA is less than 
or equal to 0.05 (see Brown, 2006). 

Table 5. Fit Indices for the Revised Model With the 2011 Senior Sample 


Cohort 

CFI 

TLI 

RMSEA 

CFit 

SRMR 

Seniors 

.953 

.944 

.038 

1.00 

.037 


Table 6 summarizes the differences in how items load on each factor in each of the models. For example, 
ACADPR01 was a variable included in the original five-factor NSSE model and was proposed to load on Level 
of Academic Challenge. However, in the revised six-factor model, this variable did not load on any of the factors. 
ANALYZE was another variables included in the Level of Academic Challenge scale on the original model; 
however, on the revised model, it loaded on the Higher Order Thinking Skills factor. 


Table 6. Comparison of factor loadings - Original NSSE Model and Revised Model 


Item 

Original model factor 

Revised model factor 

ACADPR01 

Level of Academic Challenge 

— 

ANALYZE 

Level of Academic Challenge 

HO Thinking Skills 

APPLYING 

Level of Academic Challenge 

HO Thinking Skills 

ENVSCHOL 

Level of Academic Challenge 

— 

EVALUATE 

Level of Academic Challenge 

HO Thinking Skills 

READASGN 

Level of Academic Challenge 

— 

SYNTHESZ 

Level of Academic Challenge 

HO Thinking Skills 

WORKHARD 

Level of Academic Challenge 

— 

WRITEMID 

Level of Academic Challenge 

Writing Challenge 

WRITEMOR 

Level of Academic Challenge 

Writing Challenge 

WRITESML 

Level of Academic Challenge 

Writing Challenge 

CLASSGRP 

Active & Collaborative Learning 

— 

COMMPROJ 

Active & Collaborative Learning 

— 

CLPRESEN 

Active & Collaborative Learning 

— 

CLQUEST 

Active & Collaborative Learning 

Student-Faculty Interaction 

OCCIDEAS 

Active & Collaborative Learning 

— 

OCCGRP 

Active & Collaborative Learning 

— 

TUTOR 

Active & Collaborative Learning 

— 
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FACFEED 

Student-Faculty Interaction 

— 

FACGRADE 

Student-Faculty Interaction 

Student-Faculty Interaction 

FACIDEAS 

Student-Faculty Interaction 

Student-Faculty Interaction 

FACPLANS 

Student-Faculty Interaction 

Student-Faculty Interaction 

FACOTHER 

Student-Faculty Interaction 

Student-Faculty Interaction 

RESRCH04 

Student-Faculty Interaction 

Student-Faculty Interaction 

COCURR01 

Enriching Educ Experiences 

— 

DIFFSTU2 

Enriching Educ Experiences 

Diversity 

DIVRSTUD 

Enriching Educ Experiences 

Diversity 

ENVDIVRS 

Enriching Educ Experiences 

Supportive Campus Environment 

FORLNG04 

Enriching Educ Experiences 

— 

INDSTD04 

Enriching Educ Experiences 

— 

INTERN04 

Enriching Educ Experiences 

— 

ITACADEM 

Enriching Educ Experiences 

— 

LRNCOM04 

Enriching Educ Experiences 

— 

SNRX04 

Enriching Educ Experiences 

— 

STDABR04 

Enriching Educ Experiences 

— 

VOLNTR04 

Enriching Educ Experiences 

— 

ENVACAD 

Supportive Campus Environ 

Supportive Campus Environment 

ENVADM 

Supportive Campus Environ 

Quality of Relationships 

ENVFAC 

Supportive Campus Environ 

Quality of Relationships 

ENVSOCAL 

Supportive Campus Environ 

Supportive Campus Environment 

ENVSUPRT 

Supportive Campus Environ 

Supportive Campus Environment 

ENVSTU 

Supportive Campus Environ 

Quality of Relationships 


Note: Refer to the College Student Report NSSE 2008 and 2011 Codebook for the actual questions 
(www.nsse.iub.edu/). 

5. Discussion and Conclusion 

The National Survey of Student Engagement (NSSE) was created to determine the degree to which 
postsecondary students engage in meaningful educational practices that will enhance their educational outcomes. 
NSSE provides summative information on five scales that can be used to compare institutions or cohorts within 
institutions. The use of NSSE has increased dramatically among U.S. colleges and universities and is now an 
important component of the Voluntary System of Accountability, enhancing the potential high stakes use of this 
instrument. Because other studies have noted that the NSSE may not be appropriate to measure engagement at a 
particular institution, this study was conducted to examine how well the five-factor NSSE model fit the 2008 
seniors’ data from a research intensive institution. As has been found by others within their own institutional 
contexts (i.e., Campbell & Cabrera, 2011; Gordon, Ludlum, & Hoey, 2008; LaNasa et al., 2009), the results of a 
Confirmatory Factor Analysis indicated that the model did not fit the data at our institution. Consequently, an 
alternative model was developed by conducting an exploratory factor analysis on the 2008 data that resulted in a 
six-factor structure. This revised model was then used in a CFA to test its fit with the 2011 seniors’ data at the 
same institution. These results indicated that the new model fit the data better than the original five-factor NSSE 
model. 

Table 6 provides a comparison of how each of the original 42 items were intended to load on the original five 
factors of the NSSE model and how they loaded on the revised six-factor model. Some items loaded on the 
similarly titled factor in each model. For instance, FACGRADE and FACIDEAS both loaded on factors labeled 
“Student-faculty interaction,” Other items, such as DIFFSTU2 and DIVERSTUD, loaded on the original NSSE 
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factor labeled “Enriching educational experiences.” In our six-factor model, these two variables loaded on a 
more specific aspect of those experiences: “Diversity.” Similarly, ANALYZE and APPLYING loaded on the 
original NSSE factor of “Level of Academic Challenge.” In our model, these items loaded on “Higher Order 
Thinking Skills.” 

In their study at an urban university of first-year students’ data, LaNasa et al. (2009) found a model with eight 
factors best fit those students. This eight-factor structure differs from our six-factor structure derived from 
seniors’ data. Though there are similarities between the models in the two studies, the focus of each of our six 
factors was different and so was differently labeled. For instance, “learning strategies,” one of the LaNasa et al., 
factors, had four items from the original NSSE factor labeled “Level of Academic Challenge:” (1) synthesize; (2) 
analyze; (3) evaluate; and (4) applying. In our study a similar factor was labeled “higher-order thinking skills.” 
Likewise, they have a factor labeled “institutional emphasis” and the four items that measure that factor were 
part of the “supportive campus environment” factor in our study. Both models - ours and the one developed by 
LaNasa et al. - provided more focused factors (e.g., “learning strategies” or “higher-order thinking skills”) than 
the more generic factors (e.g., “level of academic challenge”) of NSSE. Differences in our model and that of 
LaNasa et al. may be due to the missions of the different universities - theirs being urban and research-intensive, 
while ours is a public, land-grant, research-intensive university (that is also located in a more semi-rural area). 
Differences might also be due to their use of first-year students and our use of senior students. We posit that the 
information collected by the NSSE instrument should be primarily of use to institutional stakeholders and when 
there are commonalities across institutions then comparisons may be fruitful. However, it is important to 
consider differences in student bodies. For example, though NSSE is administered to both freshman cohorts and 
senior cohorts at each institution, we chose to examine only the senior group at our university, assuming that 
levels and types of engagement would be significantly different between those two groups. In fact, in 
preliminary comparisons between these two groups, seniors, as would be expected, had higher mean scores, 
indicating higher engagement levels, than first-year students. It may be that some institutions may wish to 
develop or test a model of engagement using freshman data as a way to present formative evaluations of the 
college experience - addressing potential changes that might be indicated through the freshman data to enhance 
engagement as student proceed through the system to their graduating year. 

Influencing decision making is one of the hardest challenges in assessment (Pike, 2002). Pike underlined the 
importance of practical decisions that could be made based on the assessment results. In this study, based on the 
strong model-data fit that was obtained, it is possible to conclude that this university should place special 
emphasis on the six factors—student-faculty interaction, higher-order thinking skill, supportive campus 
environment, quality of relationship, writing challenge, and diversity—as a way to examine student engagement 
at the institution. Doing so may positively impact student engagement and their educational experiences at this 
institution. 

6. Limitations 

The scope of this study was limited to data from one institution and from only the senior group of students at that 
institution responding to the NSSE. Response rates to the survey were typical of response rates at other 
institutions but still low (below 35%), with no follow-up of no respondents to examine potential sources of 
response bias. Processes and perceptions of student engagement may be different from institution to institution, a 
finding suggested by the results of this study. Engagement may also vary between seniors and freshmen at any 
institution. 

In order to strengthen the findings of this study, it would be helpful to have similar studies replicated at other 
colleges and universities that participated in the NSSE survey. Unlike Swerdzewski et al. (2007), Lutz and 
Culver (2010), and Campbell and Cabrera (2011), studies at the individual institution level should go beyond 
determining whether or not the five-factor model fits their data by proceeding to develop an alternate set of 
factor structures that may better represent their data. Such replication studies can contribute to the overall 
discussion and improvement of the NSSE survey so that meaningful interpretations and comparison among 
schools can be drawn from the NSSE data. 
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